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Abstract: Although many programs remotely disseminate information to students about the 
college application process, there is little evidence as to how students experience these programs. 
This paper examines a large-scale remote counseling program in which college counselors 
initiated interactions with 15,000 high school seniors via text message to support them through 
the college application process. Given the passive nature of text messaging, not all of the 
counselors' prompts elicited similar responses from students. I use text-as-data methods 
(combining qualitative coding and supervised machine learning) to measure which interactions 
lead to productive engagement between counselors and students, and which do not. I show that 
interactions about financial aid offers and financial aid applications are much more likely to 
generate productive engagement than interactions about college lists. This finding may help to 
explain why recent remote counseling interventions that have sought to influence students' 


college lists have been ineffective. 
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Although most students aspire to attend college, many cannot successfully navigate the 
college application and enrollment process. This is particularly true for low income students. 
Prior studies have shown that one-on-one college advising can provide a crucial support for 
students during this process and can increase the chances that students enroll and persist in 
college, but programs that have tried to scale advising through informational campaigns and text 
messaging have had mixed results. Because prior studies have focused on whether each program 
works, instead of why each program works, we still know little about how to design an effective, 
scalable counseling program. 

This study focuses on the why and how of college advisement. In particular, I provide 
new insights into the mediators of successful texting-based college advising by examining the 
detailed text-based engagement between college counselors and students in a large-scale study. I 
analyze a program in which remote counselors engaged with 15,000 low- and middle-income 
high school seniors who were on-track for college in 2016-17. Counselors initiated conversations 
with students on topics such as constructing college lists, submitting financial aid applications, 
understanding financial aid offers, and completing any necessary summer steps before fall 
enrollment. Given the passive nature of text messaging, not all messages elicited responses from 
students equally. I evaluate which of these topics are most likely to generate a 1) response, 2) 
repeated response, and 3) productive engagement between students and counselors that allows 
students to progress in the college application process. Instead of taking the information 
counselors disseminate to students as given, this analysis specifically discovers which topics and 
types of information are timely and useful to students as they apply to college. 

This paper has three primary contributions. First, this is the first application of text-as- 


data methods to college advising. Previous large quantitative studies have not analyzed student- 
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counselor interactions due to the time constraints associated with reading potentially hundreds of 
thousands of messages, but text-as-data methods can expand the number of observations that are 
possible to analyze by teaching machine learning algorithms to replicate human coding (Fesler et 
al., 2019; Grimmer & Stewart, 2013). 

Second, this paper provides guidance to the designers of remote counseling programs as 
to which types of messages are most likely to generate a productive engagement. Prior to this 
intervention, researchers and policymakers have pushed for students to focus on their college 
lists to increase their chances of enrolling in a more selective school (Hoxby & Turner, 2013). 
However, my results show that students are more likely to productively engage about financial 
aid offers (15 percent of students) and financial aid applications (13 percent of students) than 
college lists (2 percent of students). The text interactions reveal that students were receptive to 
learning new information from counselors about their financial aid offers and applications but 
were less open to receiving advice from counselors about which schools would be good fits for 
them. 

Third, methodologically it demonstrates how text-as-data methods can measure 
engagement with an intervention. Text messaging is an increasingly popular method used by 
organizations and universities (Castleman & Page, 2015, 2017; Fryer, 2016; Oreopoulos et al., 
2018; Page & Gehlbach, 2017; York et al., 2019), and more of these programs could analyze 
student text responses to understand which elements of the program are working for students. 
Text-as-data methods allow for the analysis of large amounts of text data without being 


prohibitively time consuming, and thus open the doors to many new types of analysis. 
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College Application Process 
Application Steps 

Even though many students aspire to attend college, many students do not successfully 
complete any college applications (Klasik, 2012). Considering the length and complexity of the 
college application process, this is not surprising. Students need to construct a list of schools they 
would be interested in attending, take any required entrance exams (like the SAT or ACT), fill 
out applications for each of those schools, make sure their transcripts and entrance exam scores 
are sent to their colleges of interest, fill out the Free Application for Federal Student Aid 
(FAFSA) and any state-specific financial aid forms, respond to any requests for financial aid 
verification, read and understand any financial aid offers they receive, make a decision about 
which college to attend considering their financial aid packages and personal preferences, and 
then complete any necessary summer steps for the college (like taking placement tests, 
submitting immunization forms, and signing up for new student orientation). 

Students struggle with each part of this process. Studies show that students have trouble 
with finishing their applications (Avery & Kane, 2004), submitting their FAFSAs (Bettinger et 
al., 2012), responding to financial aid verification requests (Cochrane et al., 2010), understanding 
their financial aid offers (Burd et al., 2018; Marx & Turner, 2018), and completing any necessary 
summer steps before enrolling (Castleman & Page, 2014). 

This process is even less navigable for low-income students. Low-income, high- 
achieving students who aspire to attend college are less likely to apply to academically matched 
schools than their high-income, high-achieving counterparts (Hoxby & Avery, 2012). Low- 


income students’ college attendance is also more reliant on their successful FAFSA submission, 
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and they may also need to complete additional financial aid verification steps that can impact 
their eligibility for the Pell Grant (Evans et al., 2017). 

Several theories illuminate why many students may struggle with the college application 
process. Students may not sufficiently optimize for long-term outcomes, and low-income 
students may have more competing pressures during their senior year than high-income students 
(including work and family responsibilities) (Thaler & Sunstein, 2008). They thus may be more 
focused on their short-term obligations over their longer-term investment in their future 
(Castleman & Page, 2015). Low-income students may also lack the type of social capital 
necessary to be aware of all of the components of the college application process and the long- 


term benefits of a college education (Bourdieu, 1986; Dika & Singh, 2002; Perna, 2006). 


College Counseling and Support 

College counselors can serve as a crucial support for students during this complicated 
process. Prior research has shown that access to college counselors increases the chances that 
students apply to and enroll in college (Castleman & Goodman, 2018; Hurwitz & Howell, 2013). 
However, many students do not have the access to college counselors that they need. The 
average public-school counselor is responsible for 482 students, which is twice the 
recommended counselor-to-student ratio of 250 to 1 (Avery et al., 2014; Clinedinst & Koranteng, 
2017). Low-income students also have less access to college counselors than higher-income 
students. In a study conducted in Boston public schools, only 17 percent of students who 
attended a lower-income school met with a guidance counselor at least four times, as compared 
to the 55 percent of students in a higher-income school (Avery & Kane, 2004). And only 28 


percent of public schools have a full-time college counselor, as compared to 49 percent of private 


Opening the Black Box of College Counseling 5 


schools (Clinedinst & Koranteng, 2017). Many low-income students instead are forced to 
conduct research on their own (sometimes only including schools who have sent them mail or 
who they have seen on television), or with help from their parents (who may have little 
knowledge of the college application process) (Roderick et al., 2008). 

Ideally, low-income students would have access to additional in-person college 
counselors, but many public high schools are resource constrained and are unable to hire 
additional counselors (Perna et al., 2008). In this context, many programs have emerged to 
provide students with information about the college application process outside of their high 
school’s college counseling program (Bettinger et al., 2012; Castleman & Goodman, 2018; 
Hoxby & Turner, 2013). Bottom Line and College Advising Corps both provide advising to high 
school seniors to develop their college lists, complete their college and financial aid applications, 
and make their college decision (Bettinger & Evans, 2019; Castleman & Goodman, 2018). 
Expanding College Opportunities provided students with brochures that contained information 
on the college application process and colleges’ net costs (Hoxby & Turner, 2013). Realize Your 
College Potential sent text message reminders, virtual advising, and small financial incentives to 
students to support them in the college application process (Gurantz et al., 2018). 

Some of these college advising programs have led to increases in college enrollment and 
graduation, and others have not. Thus far, it has been difficult to assess what makes some of 
these programs effective and others ineffective. In large part, this difficulty is due to a lack of 
information about how students receive and respond to additional support from counselors. More 
information about how students process any support from counselors in real time is crucial, 


including whether students productively engage with the content that counselors give students. 
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New text-as-data methods allow researchers to analyze hundreds of thousands of 
interactions between counselors and students for the first time, including methods that utilize 
machine learning to code virtually unlimited amounts of text data. These methods have been 
used in economics, political science, and sociology, and are starting to be used more within 
educational contexts (Fesler et al., 2019). 

In this paper, I apply novel text-as-data methods to understand when a counselor-initiated 
interaction leads to a productive engagement between the counselor and student. This contributes 
to the literature on college counseling programs and is the first study to analyze how students 
respond to and interact with college counselors at scale. This paper intends to demonstrate the 
ways in which remote counselors can productively support students in the college application 
process, as well as demonstrate how text-as-data methods can measure program engagement 


across a variety of contexts. 


Remote Counseling Program 

Four remote college counselors from a non-profit college advising organization (on 
behalf of the College Board) sent text messages to 15,000 low- and middle-income high school 
seniors who were on-track for college in 2016-17. Counselors sent ten broadcasts (approximately 
once per month) to students to initiate conversations about college lists, financial aid 
applications, the financial aid verification process, the college decision, financial aid offers, and 
summer steps. Students were divided into twelve batches, so that about 1,250 students received 
messages each school day. Real counselors responded to all students who replied to each 
broadcast, regardless of whether the student wanted to talk about the broadcasted topic or 


another relevant topic. 
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Sample 

To be included in the program, students indicated on the SAT that they were willing to be 
contacted by the College Board via text message. All students were on-track for college (i.e. 
were between the 50" and 90" percentiles in the national SAT distribution), lived in one of eight 
states (CA, FL, MA, MI, NC, NY, PA, and TX), and were identified as low- or middle-income. 
PSAT and SAT questionnaire data either do not ask for income levels or may be subject to non- 
response, thus limiting the ability to accurately identify students who are likely to enter college 
with financial need. To handle this, we relied on two approaches. First, we considered students to 
be low-income if they received a College Board SAT fee waiver. Eligibility for fee waiver status 
could occur through a variety of methods, most commonly National Student Lunch Program 
eligibility, receipt of public assistance, or participation in an authorized program serving low- 
income students (e.g., Upward Bound).! As these qualifications rely on students sharing this 
potentially sensitive information with their school counselors, not all low-income students who 
would qualify for a fee waiver are identified. The College Board supplements fee waiver 
information by developing a methodology to identify low- and middle-income students through 
an algorithm that includes student self-reported data on the SAT’s student data questionnaire 
(SDQ), high school attended, and census tract. Low-income students were identified then by 


either receipt of an SAT fee waiver or an estimated annual income below approximately 


' Students are eligible for fee waivers if they: enrolled in or eligible to participate in the National School Lunch 
Program (NSLP); the student’s annual family income falls within the Income Eligibility Guidelines set by the 
USDA Food and Nutrition Service; enrolled in a federal, state, or local program that aids students from low-income 
families (e.g., Federal TRIO programs such as Upward Bound); were receiving public assistance; lived in federally 
subsidized public housing or a foster home; are homeless, a ward of the state, or an orphan. 
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$58,000; moderate-income students were identified based on incomes below approximately 


$77,000 per year, but above the low-income threshold. 


Program Objectives 

The counselors were trained to aim for two primary objectives. First, they encouraged 
students to apply to at least six colleges, and balance their applications across a range of reach, 
match, and safety schools. (Schools are a “reach” if the student’s SAT score is below the 
school’s 25" percentile or less than 20 percent of applicants are admitted to the college, a 
“match” if the student’s SAT score is between the 25" and 75" percentiles of the school’s SAT 
scores, and a “safety” if the student’s SAT score is greater than the 75" percentile of the school’s 
SAT scores.) Second, they aimed to increase the number of students who enrolled in an 
academic fit college (which involves applying to academic fit schools, completing the 
applications to those schools, completing the financial aid process for those schools, and 
completing any necessary summer steps before enrolling). 

Counselors received specific training about how to steer students towards completing 
applications to a range of colleges. For example, counselors would encourage students to apply 
to additional schools if students stated that they only had one or two schools on their college list. 
Counselors would also ask students additional questions about other parts of the process if they 
had already completed a given step. If, for instance, students had already had a well-rounded 
college list, counselors might also inquire about the status of students’ applications. In addition, 
counselors were trained on how to respond to specific questions about FAFSA, how to read and 
interpret financial aid offers, and the most common steps students had to complete between 


accepting the admission offer and enrolling in the fall. 
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Counselor Broadcasts 

Each counselor broadcast was intended to start a conversation about a particular range of 
topics, including college lists (broadcast 1), financial aid applications (broadcasts 2-4), financial 
aid offers and bills (broadcasts 5, 8, and 9), the college decision (broadcast 6), and summer steps 
(broadcast 7). Table 1 shows the language included in each broadcast. 

Insert Table 1 here. 

The first broadcast asked students if they had a college list and was sent between 
November and December. In this exchange, counselors’ primary objective was to determine 
whether students had a balanced college list, and if not, to encourage students to apply to 
additional schools. Counselors reminded students of the application deadlines for each of the 
schools in which they expressed interest, inquired if students had questions about their college 
applications, and helped students understand how to use any fee waivers. 

The second, third, and fourth broadcasts initiated conversations about the FAFSA, Texas 
Application for State Financial Aid (TASFA, for students who live in Texas are not eligible for 
FAFSA), or the Dream Act application (for students who live in California and are not eligible 
for FAFSA), and were sent between December and March. The second broadcast asked students 
if they had submitted their FAFSA, TASFA, or Dream Act application yet. Counselors 
commonly provided information to students about how to create a Federal Student Aid (FSA) ID 
to start the FAFSA and responded to specific questions students had about their FAFSA 
application (e.g. not knowing what number to fill in for a question or having technical difficulties 
on the site). They also encouraged students to apply for additional scholarships. The third 


broadcast followed up on the FAFSA, TASFA, or Dream Act application. If students told 
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counselors they completed their FAFSA in the previous interaction, the broadcast asked students 
if they had received financial aid verification steps. Otherwise, the broadcast again inquired if 
students had completed their FAFSA yet and encouraged them to complete it as soon as possible. 
If students told counselors they completed their FAFSA in either the second or third broadcast, 
the fourth broadcast asked students if they had checked their Student Aid Report (SAR) in their 
FAFSA to verify that all of their colleges were listed. Otherwise, the broadcast again reminded 
students to submit their FAFSA, TASFA, or Dream Act application. 

The fifth broadcast asked students if they had received financial aid offers from the 
colleges to which they were accepted and was sent in March and April. In this interaction, 
counselors were trained to help students find their financial aid offers in their online portals, help 
them call the admissions office if they are missing an offer, help them understand their offers, 
and help them consider the affordability of the college. Counselors also offered to explain all of 
the amounts on students’ financial aid offers and estimate their total cost of attending the college 
if students sent in screenshots of their offers to counselors. 

The sixth broadcast asked students if they had made a college decision and was sent in 
April and May. If not, counselors offered to serve as a resource to help students think through 
their decision. 

The seventh broadcast asked students if they were aware of which steps they needed to 
take after accepting an admission offer in the spring and enrolling in the fall and was sent in May 
and June. Counselors encouraged students to sign up for new student orientation, submit any 
necessary immunization forms, and take any required placement tests. They would also point 


them to their institution-specific checklist online. 
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The eighth and ninth broadcasts asked students about their upcoming bill and were sent in 
June through August. The eighth broadcast asked if students had seen their bill yet. If not, 
counselors could help students find their bill online, or contact the institution to inquire about 
when they would be able to see it. Counselors would also help students understand the amounts 
on their bill given their financial aid offer, and help students consider if they wanted to sign up 
for a payment plan. The ninth broadcast asked students if they had paid their fall bill, and 
strongly encouraged them to do so before the deadline. 

The tenth broadcast asked students if they had any additional questions before the texting 
program ended and was sent in September. The counselors did not have an intended agenda for 


this conversation. 


Dashboard 

As counselors interacted with a student, they could see information about the student on 
their dashboard. They could view the student’s state and zip code, whether they were low- 
income or middle-income, their PSAT and SAT score ranges (in 200 point increments), the 
number of Advanced Placement (AP) exams they had taken, the number of free college 
application fee waivers (CAFWs) they had received from the College Board, the number of free 


66 


score sends they had received from the College Board, and the student’s “starter list” of potential 
colleges. The starter lists contained twelve colleges that were chosen based on the schools with 
the highest graduation rates for students from their county with similar SAT scores. For each of 
the twelve colleges on students’ starter lists, counselors could see the college name, state, and the 


25" and 75" percentile SAT scores for students who attend the institution. This list of colleges 


allowed counselors to personalize college suggestions to the student in the first broadcast. The 
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dashboards also allowed counselors to track their prior interactions with the student, so 
counselors could tailor each interaction based on their prior interactions with that student. 

Data include every text message sent by a counselor or student during the study, as well 
as all information on counselors’ dashboards. They also include students’ background 
characteristics, including students’ race, gender, predicted income, first-generation college status, 
SAT scores, high school GPA, and high school attended. I also connect these data to the 
Common Core of Data from the National Center for Education Statistics, which gives 


information on the urbanicity of students’ high school. 


Methods 

I measure three levels of student engagement in each student-counselor interaction: 
response, repeated response, and productive engagement. Response is measured as whether the 
student sent at least one message to the counselor after the broadcast. Repeated response is 
measured as whether the student sent a second message, beyond their initial “yes’ or ‘no’ text in 
response to the broadcast. 

Productive engagement is measured as whether students learn new information about the 
college application process, are reminded to take a step they have not yet taken or agree to make 
a change to their college application (described in more detail in Step 2 below). I measure 
whether productive engagement about any part of the college application process occurred, as 
well as whether productive engagement occurred for each given part of the process (e.g. college 
lists or FAFSA). 

To measure productive engagement, I use a text-as-data method that combines qualitative 


coding with supervised machine learning. This process comprises four steps. First, I chunk the 
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text messaging data into distinct observations (which I call “interactions”). Second, I hand code a 
random subset of those interactions for whether any productive engagement between the 
counselor and student occurred and the part of the college application process in which it 
occurred. Third, I build a supervised machine learning model that learns the relationship between 
the content of the text messages and whether productive engagement occurred in each message. 
Fourth, I use the machine learning model to predict whether productive engagement occurred in 
the remaining text messages that I did not hand code. I describe each of these steps in more detail 


below. 


Step 1: Divide text data into interactions 

I divide the text messages into distinct observations, which I refer to as interactions. A 
given interaction includes all text messages exchanged between counselors and students after one 
counselor broadcast and before the next broadcast, which will ensure that all relevant back-and- 
forth between counselors and students in a given conversation are analyzed together. These 
interactions usually take place over a couple of hours up to a couple of days. I have 18,032 


interactions in total (and 387,000 individual text messages). 


Step 2: Hand code subset of interactions 

I conduct a content analysis to determine the instances in which students and counselors 
could productively engage. I used an inductive coding process, in which two coders read through 
a subset of interactions to identify the most common ways that productive engagement could 
occur and the different parts of the college application process in which productive engagement 


could occur. The coders identified that productive engagement occurred when 1) students 
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learned new information about the college application process, 2) students were reminded to take 
a step they had not yet taken, or 3) students agreed to make a change to their college application 
(each of which are described in further detail below). The coders also identified six primary parts 
of the college application process in which productive engagement could occur. These included: 
1) college list, 2) college application, 3) financial aid application (primarily FAFSA), 4) financial 
aid offer, and 5) summer steps. (We also originally coded for college decision, but later dropped 
this from our analysis since it occurred so infrequently in our data.) These parts of the process 
are also aligned with prior literature on the college application process. (See Appendix A for the 
codebook.) 

First, interactions are coded as including productive engagement if students learn new 
information. Examples of this include when students ask a question (e.g. “where can I find my 
financial aid offer?”’) that counselors answer, students state something that is incorrect that 
counselors respond to with correct information (e.g. “I don’t think that I'll take out the Pell grant 
because my family doesn’t want loans”’), or students send a screenshot of a financial aid offer 
that counselors help them interpret. Those three sample interactions would all be coded as 
including productive engagement (about financial aid offers). If counselors ask students a 
question that they already know the answer to, that doesn’t count as productive engagement (e.g. 
if a student responds “yes” to “Have you found your financial aid offer?”’). 

Students are also coded as productively engaging if they are reminded to take a step they 
have not yet taken. For example, if counselors encourage a student to submit their FAFSA after 
they state that they have not yet done so, that is coded as productive engagement (about FAFSA). 
But if the student states that they have already submitted their FAFSA, no productive 


engagement about financial aid has occurred. Even if there are multiple messages back-and-forth 
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(e.g. in which the counselor confirms that the student has submitted FAFSA, completed any 
verification steps, and checked their Student Aid Report for their FAFSA), if the student has 
already successfully completed each of those steps there is no productive engagement. Note that 
in instances in which students agree verbally to work on their FAFSA that week, I cannot 
measure whether students actually log into FAFSA that week or not. Instead, I am measuring 
whether students are receptive to the information (i.e. are responsive to the counselor) and 
whether the interaction has the potential to help students along with the process. 

Students are also counted as productively engaging if they agree to make a change to 
their college application. This most typically occurs with college lists. If, for instance, the 
counselor suggests that the student apply to a particular school to balance their college list and 
the student agrees, that would count as productive engagement (about college lists). However, if 
the student does not agree to add it to their college list (stating, for example, that the school 
would be too close to home for them), then that is coded as no productive engagement. Again, I 
do not measure whether students actually change their college lists or not. I am instead 
measuring whether students are open to counselor feedback and consider adding additional 
schools to their list. 

An interaction only needs to contain one instance of productive engagement to be 
counted as including productive engagement and can include multiple forms of productive 
engagement in the same interaction. For instance, if students state that they have already found 
their financial aid offer (no productive engagement), but then text a screenshot of their financial 
aid offer for help understanding their financial aid package (productive engagement), that entire 
interaction is counted as containing productive engagement (about financial aid offers). If in that 


same interaction, students also ask about how to sign up for their college’s orientation, then that 
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interaction will be coded as including productive engagement about summer steps in addition to 
including productive engagement about financial aid offers. We thus end up with six separate 
variables that indicate whether productive engagement occurred in each of five categories, in 
addition to whether any productive engagement occurred. 

I randomly selected a subset of the interactions for two researchers to code to measure 
inter-rater reliability. Table 2 shows Cohen’s kappa for each of the six codes, and tests whether 
kappa is significantly above the threshold of 0.65 using a simulated data method. This method 
generates many datasets with a kappa below 0.65, then determines whether less than five percent 
of the samples have a kappa greater than the observed kappa (Shaffer, 2017). All of the codes 
have kappas above 0.83 and are statistically significantly above the threshold of 0.65. The coders 
also coded some messages separately, leading to a total of 551 hand-coded interactions. 


Insert Table 2 here. 


Step 3: Build a supervised machine learning model 

The hand-coded messages serve as training data for a supervised machine learning 
model, which learns the relationships between the text of the interactions and whether different 
forms of productive engagement took place (the six productive engagement codes). I then use 
this model to predict the codes of the 17,481 interactions that I have not hand coded. This allows 
me to know whether productive engagement occurred for all 18,032 interactions. 

The first step in building this model is to convert my text data into a quantitative dataset. 
I simplify my text data by using the “bag-of-words” method, in which I discard information 
about word order, punctuation, and capitalization. This method also does not differentiate 


between words texted by counselors versus words texted by students. To simplify the text 
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further, I remove word suffixes (i.e. stem the terms) and drop terms that occur in fewer than 2 
percent of messages. I then convert this processed text into a document-term matrix, in which 
each interaction is a row and each column represents a (processed) term. Each cell represents the 
number of times a given term occurred in a given interaction. 

I apply seven different machine learning algorithms to learn the relationship between the 
content of the interactions (as measured in my document-term matrix) and whether productive 
engagement took place. I assess the performance of each of the methods to determine which 
method to use to generate the most accurate predictions of which interactions include different 
forms of productive engagement. 

I use the supervised machine learning techniques elastic net, ridge, LASSO, random 
forests, support vector machine (SVM), neural nets, and an ensemble of the six methods. Elastic 
net, ridge, and LASSO are all types of regularized regression, in which the ordinary least square 
parameter estimates are shrunk towards zero to improve predictive performance (Grimmer et al., 
2019). Random forests combine multiple decision trees, and automatically choose interaction 
terms to include while optimizing for out-of-sample performance through a penalty parameter 
(Mullainathan & Spiess, 2017). Neural networks model the outcome as a nonlinear function 
based on linear combinations of the predictors, and perform much better than ordinary least 
squares when there are many predictors (Hastie et al., 2009, Chapter 11; Jurafsky & Martin, 
2018, Chapter 7). SVM generates optimal separating hyperplanes to separate different outcomes, 
and is the most widely used (and often best performing) classifier in social science (Hopkins & 
King, 2010). I also estimate an ensemble of the six methods, by estimating a constrained 


regression of the labels on the six predicted codes from the six methods (constraining the 
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coefficients to be positive and sum to one). Ensemble methods have been found to often perform 
better than any individual method (Athey et al., 2019). 

I assess the performance of each of these methods to determine how accurately each are 
able to predict out-of-sample outcomes for each of my six productive engagement outcomes. I 
thus split my hand-coded data into a training set (80% of my data) and a test set (20% of my 
data). The training set is used to build the machine learning models, and the test set is used to 
assess how accurately each of the models predict outcomes using data not included in the 
training of the model. This separation of training and test set is crucial to ensure that the machine 
learning models are not overfit to the data used to build the model. I use two common metrics 
used to assess model performance: sensitivity and specificity. Sensitivity is the proportion of true 
productive engagement that is correctly coded by the algorithm as productive engagement, and 
specificity is the proportion of true non-productive engagement that is correctly coded by the 
algorithm as non-productive engagement. 

Table 3 shows the sensitivity and specificity of the seven machine learning methods for 
the overall productive engagement category (which is one of the six predicted outcomes). Each 
of the methods performs relatively well, but SVM outperforms all of the other methods. One 
hundred percent of the interactions that contain any productive engagement in the test sample 
were correctly categorized as containing productive engagement, and 95 percent of the 
interactions that did not contain any productive engagement were correctly categorized as not 
containing productive engagement (see Table 3a). I use SVM for the remainder of my analyses. 
Table 3b shows the sensitivity and specificity of SVM for each of the productive engagement 
topics. The codes each have very high specificities (> 94%), which indicates that I predict which 


interactions include no productive engagement with very high accuracy. The sensitivities of the 
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codes average 83 percent, which indicates that I also predict which interactions include 
productive engagement with high accuracy. 
Insert Table 3 here. 

To ensure that any inaccuracies in my predictions do not bias the overall estimates of the 
proportion of students who productively engage with counselors, I use the Hopkins-King (2010) 
adjustment. This adjustment uses the sensitivity and specificity of a model to adjust the 
proportions to generate approximately unbiased and statistically consistent proportions. I find 
that these adjustments have a negligible impact on the proportions, and thus I show the 
unadjusted proportions in the rest of the paper. Appendix B provides a full description of these 
adjustments. 

Step 4: Predict productive engagement for remaining interactions 
Lastly, I use SVM to predict whether the six types of productive engagement occurred in 


the remaining 17,481 non-coded interactions, which gives me interaction-level outcome data. 


Results 
Response and Repeated Response 

Seventy one percent of the 14,860 students respond to at least one counselor broadcast, 
and 49 percent of students repeatedly respond to at least one broadcast (see the last row of Table 
4). Students are the most likely to respond to broadcasts about the FAFSA application (36 
percent of students), the college decision (30 percent of students), FAFSA verification (27 
percent of students), and the college list (26 percent of students). Students are the most likely to 
repeatedly respond to counselors after the broadcast about the college decision (20 percent of 


students) and about the college list (13 percent). 
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Insert Table 4 here. 


Productive Engagement 

Twenty-seven percent of the 14,860 students productively engage with the counselor on 
at least one topic. Students are most likely to productively respond after broadcasts about the 
FAFSA application (8 percent of students), the financial aid offer (7 percent of students), and the 
college decision (7 percent of students) (see the rightmost column in Table 4). However, these 
percentages do not tell us about which topics students productively engaged. Students and 
counselors could both re-steer the conversation if the student needed more help with another part 
of the application process than the broadcasted topic. Thus, this table also breaks out productive 
engagement by the topic about which students and counselors actually productively engaged. 

Throughout the program, 13 percent of students productively engaged about the FAFSA 
and 15 percent of students productively engaged about the financial aid offer (see the bottom row 
in Table 4). Much of the productive engagement about FAFSA occurred after the FAFSA 
broadcasts: 5.5 percent of students productively engaged about FAFSA after the FAFSA 
application broadcast, 4.5 percent after the FAFSA verification broadcast, and 3.9 percent after 
the FAFSA Student Aid Report broadcast. The productive engagement about the financial aid 
offer began after the financial aid offer broadcast (5.5 percent of students), but stayed almost as 
high through the college decision broadcast (4.7 percent of students), the finding bill broadcast 
(4.4 percent of students) and the paying bill broadcast (2 percent of students). In fact, the college 
decision broadcast led almost entirely to discussions about the financial aid offer, and to a lesser 


extent, FAFSA, as opposed to about the college decision. (The percent of interactions that 
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included productive engagement about the college decision was so close to zero that I had to 
exclude it from the analysis.) 

In contrast, only 1.8 percent of students ever productively engaged with counselors about 
their college lists. Almost all of these students productively engaged after the college list 
broadcast. A similar percent of students (1.7 percent) productively engaged about the college 
application, which also almost always happened after the college list broadcast. Similarly, only 
evaluating the percent of students who productively engage after the college list broadcast only 
tells part of the story, since those productive engagements were split between college list and 
college applications and included productive engagement about FAFSA as well. Thus, the true 
level of productive engagement about college lists (1.8 percent) is much lower than the level of 
productive engagement indicated by the total amount of productive engagement after the first 
broadcast (4.5 percent). 

Less than four percent of students ever productively engaged about summer steps. 
Almost all of these engagements occurred after the summer steps broadcast, indicating that this 
was not a persistent issue for students. 

Figure 1 shows productive engagement graphically over time. A small percent of students 
engage productively about their college lists at the very beginning of the process (i.e. in 
November), and many need help with their financial aid applications (from December through 
April). Many students also need help with their financial aid offers at two points: when they start 
receiving their offers in April and again when they need to pay their fall bill in August. 


Insert Figure | here. 
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Heterogeneity by Student Characteristics 

Students who respond to counselors’ messages are somewhat different than students who 
do not respond, but students who productively engage are similar across demographic 
characteristics (see Table 5). Female students are more likely to respond than male students, 
higher-achieving students are more likely to respond than lower-achieving students, and rural 
students are more likely to respond than non-rural students. Black students are less likely to 
respond than non-Black students, first generation and low-income students are less likely to 
respond to non-first generation and middle-income students, and students who live in cities are 
less likely to respond than students who do not live in cities. However, students across 
demographic groups are equally likely to productively engage (with the exception that students 
who live in towns are more likely to productively engage). This pattern demonstrates that the 
students from demographic groups that are less likely to respond are more likely to productively 
engage if they do respond. 

Insert Table 5 here. 

Despite the similarities across demographic groups in engaging in any productive 
interaction, some demographic groups are more likely to engage about particular parts of the 
college application process than others (see Table 6). Female students are more likely than male 
students to productively engage about their college applications, White students are more likely 
to productively engage about their college lists and less likely to engage about their college 
applications than non-White students, Asian students are more likely to engage about their 
FAFSAs than non-Asian students, and Latinx students are more likely to engage about their 


college applications and summer steps and less likely to engage about their college lists than 


Opening the Black Box of College Counseling 23 


non-Latinx students. Students who live in towns are more likely to productively engage about 
their college applications and their financial aid offers than other students. 


Insert Table 6 here. 


Discussion 

There are several potential reasons why students may be more likely to productively 
engage about their financial aid offers and FAFSA than about college lists, even though one of 
this program’s primary objective was to influence student’s college lists. First, it may be easier 
for counselors to provide useful information to students than to provide advice. In general, 
conversations about financial aid were intended to provide information (e.g. how to submit 
FAFSA) whereas conversations about college lists were intended to provide advice (e.g. this 
school would be a good fit for a particular student). Students may prefer that counselors support 
them through the more procedural parts of applying to college but are not as receptive to advice 
as personal as choosing a school (which is not only about academic and financial fit, but also 
about personal fit). For example, students would sometimes mention that they did not want to 
apply to a suggested school because it was too close to home or didn’t have their preferred 
major. 

Second, it may be even harder for remote counselors to give personal advice to students. 
Remote counselors, especially in a texting-only program, do not have many opportunities to get 
to know the individual student’s preferences and to build trust with the student. Thus, remote 
programs may be even better suited to providing information, as opposed to advice. 

Third, the timing of this program may have been too late to influence students’ college 


lists. This program began in late October of students’ senior year, and it is possible that students 


Opening the Black Box of College Counseling 24 


already had fairly firm ideas about where they would like to apply by then. More students may 
have productively engaged about college lists if counselors texted them about potential schools 
during their junior year, for instance. 

I also found a lower level of productive engagement about summer steps than previous 
programs. It is possible that more students would have productively engaged about their summer 
steps if the counselors were from the institution in which they enrolled, and thus could have 


given information that was even more institution-specific (as in Page & Gehlbach, 2017). 


Conclusion 

This paper shows how low- and middle-income students respond to counselor support in 
a large-scale remote counseling program. By examining the detailed text message exchanges 
between counselors and students, I find that students are less receptive to counselor advice about 
expanding their college lists to include a broader array of schools, and more receptive to concrete 
information about financial aid applications and financial aid offers. This is one of the first 
studies that focuses on how students experience large-scale advising programs, and may help to 
explain why prior studies that focus on expanding students’ college lists have had no impacts on 
college enrollment (Gurantz et al., 2018). Program designers of future text-based advising 
programs may want to focus on supporting students through the financial aid process rather than 
encouraging students to apply to more schools. 

My exploratory analysis also indicates that students from different demographic groups 
may respond to counselor advice differently, suggesting that program designers may need to 
consider their target student group when determining which types of support would be most 


useful to students in their program. 
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This paper also demonstrates how text-as-data methods can be used to measure student 
engagement with an educational program. Many programs collect text data as part of their 
regular operations, including an increasing number that specifically utilize text messaging 
(Castleman & Page, 2015, 2017; Fryer, 2016; Oreopoulos et al., 2018; Page & Gehlbach, 2017; 
York et al., 2019). These texting-based programs could utilize the supervised machine learning 
techniques demonstrated in this paper to better understand their own program engagement by 
analyzing the text data they are already collecting. These text-as-data methods could be applied 
to a much broader array of programs to allow program designers to both understand how 
participants experience the program and to determine how the program could be improved upon 


in the future. 
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Table 1: Counselor Broadcasts 


7 | Topic Date Range | Broadcast Language 
November — - 
i 2 
1 | College List eseube Do you know where you want to apply to college’ 
FAFSA December - 
i v. 
2 (Appligahien) February Have you had a chance to submit your FAFSA app yet’ 
1. Do you need to complete verification steps for your 
3 FAFSA January - | FAFSA? 
(Verification) February 2. Have you had a chance to submit your 
FAFSA/TASFA/DREAM ACT app yet? 
1. Have you checked your Student Aid Report (SAR) in your 
FAFSA (Student | February - | FAFSA to make sure all of your colleges are listed? 
4 | Aid Report) March 2. Submit your FAFSA/Dream Act/TASFA asap to meet 
deadlines! Do you know how? 
5 Financial Aid Widzehis ACT If you applied for aid, you will receive a Financial Aid Award 
Offer aes Offer from the colleges that accept you. Did you get these? 
6 | College Decision | April - May Have you decided on which college you’re going to attend? 
If you’re starting college in the fall, there are important steps 
A) URE REDE ae ee: to take over the summer. Do you know what you need to do? 
cae : You'll most likely need to pay your fall college bill in 
ener dunes PUSHE July/Aug. Have you seen your bill yet? 
9 | Paying Bill August Have you already paid your fall college bill? 
10 | Other Questions | September Do you have any other questions? 


Note: Counselor broadcasts are the initial messages that counselors text to students, that are intended 
to begin a conversation about a particular part of the college application process. 
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Table 2: Inter-Rater Reliability of Productive Engagement Codes 


Kappa 
College List 0.91 «x 
College Applications 0.88 
FAFSA 0.86%* 
Financial Aid Offer 0.83*« 
Summer Steps 0.96**« 


Any Productive Engagement 0.91 **x 


Asterisks indicate whether kappa > 0.65. 
peO10 pa 0.05 peng 
No 244 
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Table 3: Predictive Performance of Machine Learning Methods 


(a) Predictive Performance by Machine Learning Method for Any Productive Engagement 


Machine Learning Method _ Sensitivity (%) Specificity (%) 


Support Vector Machine 100 94.5 
Neural Net 92.7 70.9 
Elastic Net 54.9 100 
Ridge 56.1 92.7 
LASSO 56.1 94.5 
Random Forest 100 70.9 
Ensemble 96.3 81.8 


(b) Predictive Performance by Productive Engagement Category using SVM 


Productive Engagement Category Sensitivity (%) Specificity (%) 
College List 63.6 100 
College Applications 87.5 100 
FAFSA 92.3 94.6 
Financial Aid Offer 88.6 98.0 
Summer Steps 66.7 100 
Any Productive Engagement 100 94.5 
N = 551. 


Sensitivity is the proportion of true productive engagement that is correctly coded by 
the algorithm as productive engagement, and specificity is the proportion of true 
non-productive engagement that is correctly coded by the algorithm as 
non-productive engagement. 
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Opening the Black Box of College Counseling 
Figure 1: Productive Engagement by Month 
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Appendix A: Productive Engagement Codebook 


Not Productive Engagement 


Productive Engagement 


The student asks a question, expresses 
confusion, complains, or brings up a 
new topic related to the application 
process, to which the counselor 
responds with information or a 
suggestion. This information can be 
given as a question. 

The student states something that is 
incorrect, and the counselor corrects 
them. 


The counselor asks the student to take a 


step that they have not yet taken. 

The student realizes that they had not 
completed a step that they thought they 
had. 


The counselor provides students with 
information that the student did not 
request (or express confusion about), 
and the student indicates that this is new 
information. 


The student sends in a screenshot of 
something and the counselor helps them 
analyze it. 


The student asks a question, and the 
counselor responds with a follow up 
question that the student doesn’t 
respond to. The follow up question 
does not include any helpful 
information. 

The counselor asks students a series 
of questions, and then determines 
that the student is on the right 
track/doing the right thing. The 
student never expresses confusion or 
asks a question about the process. 


The counselor asks students to 
confirm that they have taken a step, 
and the student confirms that they 
have. 


The counselor provides students with 
information that the student did not 
request (or express confusion about), 
and the student does not 
acknowledge that this is new 
information (this includes 
nonresponse). 


The student tries to send ina 
screenshot of something, it doesn’t 
go through, and the student stops 
responding. 


Note: The interaction is coded as containing productive engagement if there are one or more 
instances of productive engagement. 
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Summary of Codes 


Code Name Code Description 


College List Choosing colleges, assessing fit with colleges, and learning more about 
particular schools or majors. 


College Application deadlines, application fee waivers, how to submit SAT/ACT 
Application scores, and how to write college essay, etc. 


Financial Aid | Finding and submitting FAFSA, TASFA, Cal Grant, and any scholarships; 
Application checking the Student Aid Report (SAR); and submitting additional 
information to FAFSA/colleges (financial aid verification). 


Financial Aid | Finding financial aid offer (through online portal, calling financial aid office, 

Offer or finding letter in the mail); explaining the offer; discussing loans, work 
study, working over the summer, and creating a payment plan with students’ 
family, etc. 


Bill Finding the bill, bill deadline, and what bill contains. 


College Deciding between multiple schools and discussing schools that the student 
Decision has already applied to/been accepted to. 


Summer Steps | Placement tests, immunization forms, signing up for orientation, etc. 
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College List 


Applying to additional schools 

Learning about the schools that the student is interested in 

Attending college while joining the military 

Chances of being admitted to a school, and academic fit (comparing SAT scores, GPA, 
etc.) 

Choosing a major, or thinking about future career options 

What schools pay attention to when making college decisions 


College Application 


Application deadlines 

Sending information to colleges: SAT/ACT scores, transcripts, essays, the Common app, 
letters of recommendation 

Application fee waivers 

Whether student needs to do anything additional once they’ve been deferred 

Finding application website 


Financial Aid Application 
Applying 


FAFSA/Cal Grant/TASFA/CSS Profile 

Financial aid deadlines 

Contacting schools’ financial aid offices to explain students’ specific financial 
circumstances before being admitted 

Student aid report (SAR) 

Finding and submitting scholarships 


Verification and fixing incomplete or incorrect financial aid offers 


FAFSA requesting verification steps 

Colleges requesting additional financial information after a student is accepted, which 
will affect a student’s financial aid award letter 

Not having all grants/scholarships listed on financial aid award letter 

Ensuring that state grants are applied to the school students want to attend 


Financial Aid Offer 
Finding Financial Aid Offer 


Student navigates to online portal to check if financial aid is there (whether it’s there or 
not). 

Call school’s financial aid office to inquire about where the financial aid award offer is 
Student asks where to find additional information about their award for the following 
year 


Understanding Financial Aid Offer 


Explaining a students’ individual financial aid award amounts 
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Understanding differences between grants and loans 

Understanding loan interest rates 

Defining terms (Parent PLUS, work study, subsidized loans, unsubsidized loans, etc.). 
This can be discussed in direct connection with an award letter, or more general 
questions. 

Understanding the loans that have been offered to students 

Accepting financial aid/loans (and deadlines for this) 

General questions about how financial aid works after the student has been admitted 


Affording College 


Bill 


Strategizing how to pay for college/creating a plan to pay 
Work study 
Taking out loans 
o Steps to take out loans (entrance loan counseling and the master promissory note, 
MPN) 
o Whether taking out loans is a good idea 
o How loans might affect credit scores 
Talking to parents about paying for college 
Working over the summer to pay for college 
Payment plans 
Talking through how to pay for rent 


Bill deadline 

What bill consists of/what a bill is 
Logging into portal to view bill 
Submitting/paying bill 

Requesting an extension for paying the bill 


College Decision 


Pros and cons of going to different schools (with different financial packages) 
Discussions about a school that a student has already applied to, but hasn’t yet accepted 
the offer 

Deciding whether to attend college at all 

Deciding whether to take a gap year 

Creating a plan to go to college in the future (not attending this year) 


Summer Steps 


How to sign up for orientation or what orientation is 

How to sign up for placement tests or what they entail 

How to fill out immunization forms 

How to create a school account (after the student was admitted) - unless the student is 
creating an account to view their financial aid award letter 

Discussions about the school the student has decided to attend 
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e Sending information to colleges after being admitted (like transcripts and AP scores) 
e Signing up for classes, and finding cheap books 
e Student asking questions about the school that they are going to attend 
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Appendix B: Proportions Adjustment 

Although SVM generates the best prediction for each individual observation, the sample- 
level proportions may be biased if the model consistently misclassifies true positives more or less 
frequently than true negatives. We can generate approximately unbiased and statistically 
consistent proportions by adjusting our proportions based on the performance in our test sample 
(Hopkins & King, 2010). The proportion of documents with a predicted positive code (D = 1) 
must be either true positives or false positives: 

P(D =1) = P® =1|D = 1)]P(D = 1) + (1-— P@ = O|D = 0))P( = 0), 

where D represents the binary prediction and D represents the true binary value. P(D = 
1|D = 1) is the sensitivity of the model, and P(D = 0|D = 0) is the specificity of the model 
(which we showed in Table 3). To estimate the true proportion, we rearrange this equation: 


P(D =1)-—(1- specificit 
sensitivity — (1 — specificity) 
I apply this adjustment to the proportions in my sample and find that the adjustments do 
not substantively change my results (see Tables B1 — B6). For simplicity, I show the unadjusted 


proportions in my main paper. 
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Table B1: Any Productive Engagement by Broadcast Topic 


Unadjusted Adjusted 


1. College List 4.5 3.7 
2. FAFSA (Application) hake 7.1 
3. FAFSA (Verification) 5.2 4.6 
4. FAFSA (Student Aid Report) 4.5 4.0 
5. Financial Aid Offer 6.8 6.2 
6. College Decision 6.7 5.6 
7. Summer Steps 5.6 4.9 
8. Finding Bill 5.4 4.8 
9. Paying Bill 3.3 2.6 
10. Other Questions 1.5 1.1 


Table B2: Productive Engagement on College Lists by Broadcast Topic 


Unadjusted Adjusted 


1. College List 1.7 let 
2. FAFSA (Application) 0.1 0.1 
3. FAFSA (Verification) 0.0 0.0 
4. FAFSA (Student Aid Report) 0.0 0.0 
5. Financial Aid Offer 0.0 0.0 
6. College Decision 0.0 0.0 
7. Summer Steps 0.0 0.0 
8. Finding Bill 0.0 0.0 
9. Paying Bill 0.0 0.0 
10. Other Questions 0.0 0.0 


Table B3: Productive Engagement on College Applications by Broadcast Topic 


Unadjusted Adjusted 


1. College List 3 13 
2. FAFSA (Application) 0.2 0.2 
3. FAFSA (Verification) 0.1 0.1 
4. FAFSA (Student Aid Report) 0.0 0.0 
5. Financial Aid Offer 0.0 0.0 
6. College Decision 0.0 0.0 
7. Summer Steps 0.0 0.0 
8. Finding Bill 0.0 0.0 
9. Paying Bill 0.0 0.0 
10. Other Questions 0.0 0.0 
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Table B4: Productive Engagement on FAFSA by Broadcast Topic 


Unadjusted Adjusted 


1. College List 0.7 0.0 
2. FAFSA (Application) 5.5 4.7 
3. FAFSA (Verification) 4.5 3.7 
4. FAFSA (Student Aid Report) 3.9 3.4 
5. Financial Aid Offer 0.9 0.3 
6. College Decision 0.7 0.0 
7. Summer Steps 0.3 0.0 
8. Finding Bill 0.5 0.0 
9. Paying Bill 0.4 0.0 
10. Other Questions 0.1 0.0 


Table B5: Productive Engagement on Financial Aid Offers by Broadcast Topic 


Unadjusted Adjusted 


1. College List 0.1 0.0 
2. FAFSA (Application) 0.2 0.0 
3. FAFSA (Verification) 0.3 0.1 
4. FAFSA (Student Aid Report) 0.6 0.4 
5. Financial Aid Offer 5.5 5.3 
6. College Decision 4.7 4.2 
7. Summer Steps 1.5 1.3 
8. Finding Bill 4.4 4.1 
9. Paying Bill 2.0 1.7 
10. Other Questions 0.5 0.4 


Table B6: Productive Engagement on Summer Steps by Broadcast Topic 


Unadjusted Adjusted 


1. College List 0.0 0.0 
2. FAFSA (Application) 0.0 0.0 
3. FAFSA (Verification) 0.0 0.0 
4. FAFSA (Student Aid Report) 0.0 0.0 
5. Financial Aid Offer 0.0 0.0 
6. College Decision 0.0 0.0 
7. Summer Steps 3.0 3.0 
8. Finding Bill 0.1 0.1 
9. Paying Bill 0:2 0:2 
10. Other Questions 0.1 0.1 


